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© IMAGE SIGNAL CONVERTER. 



© A digitally inputted image signal (SD signal) is converted Into a high resolution digital image signal (HD 
2 signal) by using estimated values. The specific pixels of an object to be estimated are classified according to the 
^ one-, two-, or three-dimension level distribution of plural reference pixels of SO signal located near the specific 
CO pixels. The estimated values of the specific pixels are generated by a linear combination of the values of the 
^ reference values and the estimation coefficients determined in advance by learning. During the learning, a known 

HD signal and the SO signal formed from it are used to determine the estimation coefficients so as to minimize 
lO the sum of squares of the en-ors between the true values and the values estimated by the linear combination of 
2 the values of the sun-ounding SD signal reference pixels and the estimated coefficients. Not necessarily limited 

to the estimated coefficients, it may be possible to use. as the estimated values, representatives determined for 
O every class in con^espondence with the class of the inputted SD signal. The reference value of a block and the 
^ value normalized by the dynamic range OR are an example as the representatives. 
lU 
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Technical Field 

The present invention relates to an image signal converting apparatus suitable for so-called up- 
conversion where an image signal with a standard resolution is converted into an image signal with a high 
5 resolution. 

Background Art 

For example, in the case of TV signals, in addition to standard resolution (or standard definition SO) TV 

10 signals, high definition (or high definition HD) TV signals have t>een experimentatly used for broadcasting. 
TV receivers that can receive HD signals have been practically used. In other words, the HO TV system is 
going to coexist with the SO TV system. 

When the SO TV system and the HO TV system coexist, a signal converting apparatus for allowing an 
HO TV monitor to reproduce SO TV video signals will be required. Although various standards for the SO 

/5 signals and the HD signals have been proposed, now assume an NTSC system for SO signals (number of 
scanning lines = 525. number of fields = 60, aspect ratio = 4:3) and an HDTV system for HO signals 
(number of scanning lines = 1 125, number of fields = 60. aspect ratio ~ 16:9). The resolution of the HO 
signals is four times higher than the resolution of the SO signals. Thus, the converting apparatus should 
perform an up-converting process that increases the number of pixels of the input SO signal four times so 

20 as to obtain the resolution of an HO signal. 

Conventionally, the signal converting apparatus uses interpolating filters. Fig. 1 shows a block diagram 
of a conventional signal up*K:onverting apparatus. An' SO signal is supplied from an Input terminal 1. The SO 
signal is sent to a horizontal interpolating fitter 2 that increases the number of pixels of the SO signal twice 
in horizontal direction. The output of the horizontal interpolating filter 2 Is sent to a vertical Interpolating filter 

25 3 that Increases the number of pixels of tiie output signal of the horizontal Interpolating filter 2 twice In 
vertical direction. The output of tiie vertical Interpolating filter 3 Is an HD signal. The HO signal Is sent to an 
output terminal 4. In the conventional signal converting apparatus, the image is up-converted with such 
filters. For example, 0 data is inserted into an interpolating point so that the sampling frequency of the SO 
signal accords with tiie sampling fr^uency of tiie HD signal. An Interpolated value corresponding to the 

30 interpolating point is formed by each filter. 

Fig. 2 shows a construction of each of the Interpolating fitters. An SO signal is supplied to an input 
terminal 5. The SO signal Is sent to multipliers that multiply the SO signal by filter coefficients an,a„-i, ... ao. 
Each of outputs of tiie multipliers is sent to a register with a unit delay amount T. The output of each of the 
multipliers and tiie output of each of the registers T are added. An Interpolated output is sent to an output 

35 terminal 6. In tiie horizontal interpolating filter 2, the unit delay amount T accords with a sample period. In 
the vertical interpolating filter 3. the unit delay amount T accords with a line period. 

In tiie conventional image signal converting apparatus, since the SO signal is up-converted into the HO 
signal with the filters, tiie output HO signal Is generated by processing the input SO signal. Thus, the 
resolution of the output HD signal is not higher tiian tiie resolution of tiie input SO signal. 

40 An object of the present invention is to provide an image signal converting apparatus ttiat can 
compensate tiie resolution of tiie output signal. 

Another object of the present invention is to provide an image signal converting apparatus that 
categorizes tiie input signal as classes corresponding to local features of the input image signal so as to 
improve conversion accuracy. 

45 

Disclosure of Invention 

A first aspect of the present invention is a signal converting apparatus for converting a first digital image 
signal into a second digital image signal having higher resolution than tiie first digital image signal, 

so comprising a class categorizing means for categorizing a considered pixel of the second digital image 
signal as a class corresponding to a pattern of level dIstritHition of a plurality of reference pixels of tiie first 
digital image signal, the reference pixels of the tirst digital Image signal t>eing disposed spatially and/or 
chronologically adjacent to the considered pixel, a memory means for storing predicted coefficients for each 
class and for outputting tiie predicted coefficients corresponding to class information received from tiie 

55 class categorizing means, and a predicted value generating means for calculating linear combination of the 
values of tiie pixels of the first digital image signal spatially and/or chronologically adjacent to the 
considered pixel so as to generate a predicted value of tiie considered pixel. The predicted coefficients are 
pre-tearnt and pre-determined. 
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To compress data, various techniques such as OCT (Discrete Cosine Transform), VQ (Vector Quan- 
tizing). OPCM (Differential Pulse Code Modulation). BCT (Block Truncation Coding), and non-linear quan- 
tizing may be selectively used as well as the ADRC (Adaptive Dynamic Range Coding) technique. 

As shown in Fig. 4. the block segmenting circuit 12 converts raster scanning sequence data of the TV 
5 signal into two-dimension block sequence data. In the example shown in Rg. 4. one block is composed of 
(3 X 3) SD pixels denoted by a to i. The predicted value generating circuit 16 generates the values of four 
HD pixels A to D nearly at the center of the block corresponding to the SD pixels of the block. 

Instead of the block shown in Rg. 4. the predicted value generating circuit 16 may generate predicted 
values of HD pixels corresponding to a one-dimensional block of four SD pixels a to d as shown in Rg, 5. 
w Moreover, the present invention may be applied to a three-dimensional block that will be descried later. 

In the one-dimensional arrangement shown in Rg. 5, the HD pixel A is categorized as a class and a 
predicted value thereof is generated corresponding to the SD pixels a. b, and c. The HD pixel 8 is 
categorized as a class and a predicted value thereof is generated corresponding to the SD pixels a. b, c. 
and d. Likewise, these class categorizing process and a predicted value generating process apply to the 
15 learning operation. 

Next, the ADRC technique used for the data compressing circuit 13 will be described. In the ADRC 
technique, redundancy of level is adaptively removed corresponding to the correlation of local features of 
pixels. As shown in Rg. 6. it is clear that dynamic ranges A and 8 in each block that is re-quantized are 
much smaller than the original dynamic range of 0 to 255 of the original data composed of eight bits. Thus. 
20 the number of bits necessary for re-quantizing these blocks is much smaller than the eight bits. 

Now. assume that the number of assigned bits that is smaller than the number of original bits (eight 
bits) is p. the dynamic range of a block is DR. the value of a pixel in the block is x. and a re-quantized code 
is Q. With formula (1). the level l>etween maximum value MAX and minimum value MIN is equally divided 
into 2** portions and then re-quantized. Rg. 7(a) shows a re-quantizing operation of a level where p is 3. 

25 

DR = MAX - MIN + 1 

Q = [(X - MIN + 0.5) X 2P / DR] (1) 

where [z] represents the maximum integer that is z or less. 
30 Next, the mean value of pixels in the block having the data level equivalent to (2^ - 1) in the p-bit re- 
quantizing graduation level in Rg. 7(a) is calculated. The cateulated result is a new maximum value MAXV 
The mean value of the pixels in the block having the data level equivalent to the re-quantizing graduation 
level 0 is a new minimum value MIN'. With the new maximum value MAX* and the new minimum value 
MIN\ the dynamic range is defined. With formula (2), the re-quantizing operation is performed. 

35 

DR' = MAX* - MIN' 

q = [(X . MIN') X (2** - 1) / DR' + 0.5] (2) 

where (zj represents the maximum integer that is z or less. 
40 In the ADRC technique where the new maximum value MAX*, the minimum value MIN'. and the 

dynamic range DR' are re-defined, information amount can t>e effectively compressed. The compressed 

result is not affected by noise. 

As a quantizing operation corresponding to the ADRC technique, representative level being restored 

may have the same level as the maximum value MAX and the minimum value MIN. 
45 As an example of a two-dimensional block, corresponding to the above-described ADRC technique. 

values of n pixels where eight bits with values a to i have been compressed to p bits are sent to the class 

code generating circuit 14. Thus, a code class representing class c is generated corresponding to formula 



50 

class = E (2^)^ (3) 



Next, with reference to Rg. 8, an ADRC circuit corresponding to the one-bit ADRC technique of will be 
described. In Rg. 8. a block sequence data is supplied to an input terminal 21. The block sequence data is 
sent to a detecting circuit 22 that detects the maximum value MAX and the minimum valOe MIN of pixels 
for each block. The MAX and MIN are sent to a subtracting circuit 23. The output of the subtracting circuit 
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The horizontal thin-out filter 43 thins out the HO signal in horizontal direction so as to reduce the number of 
pixels of the HO signal in horizontal direction by 1/2. Thus, an image signal with pixels same as an SO 
signal is sent to a learning portion 44. A predicted coefficient memory 45 stores predicted coefficients wl to 
wn at an address corresponding to the class determined by a clacs categorizing circuit in the learning 
5 portion 44. 

The method for forming an SO signal from an HQ signal is not limited to the above-described method 
where the thin-out filters are used. Instead, as in the pixel arrangement shown in Fig. 4. the value of an SO 
pixel a may be formed by the mean value of <2 x 2) pixels (for example A. B. C. and 0). Alternatively, unlike 
with such a simple mean value, a weight mean value of HO pixels in a wide range (for example. HO pixels 
10 in one block) may be used for an SD pixel. 

As in the arrangement shown in Rg. 4, when one block is composed of (3 x 3) SD pixels, the SO pixels 
a to i and the HO pixels A. B. C. and D t>ecome one set of learnt data. When there are many sets of learnt 
data for one frame and the number of frames is increased, a targe number of sets of learnt data can be 
used. 

15 The learning portion 44 performs both a class categorizing process and a calculating process. As the 
class categorizing process, the learning portion 44 compresses the SO pixels a to i and categorizes the 
pattem of the two-dimensional distribution of the values of the compressed SD pixels as a class. As the 
calculating process, the learning portion 44 determines predicted coefficients of each class corresponding 
to the method of (east squares. The class categorizing process perforrned by the learning portion 44 is the 

20 same as the process performed by the data compressing circuit 13 and the class code generating circuit 14 
shown in Rg. 3. This learning portion 44 may be accomplished by software. Rg. 12 is a ftow chart showing 
the process of software for the learning portion 44. 

At step 51. the control of the learning portion 44 Is started. At step 52 "SEGMENT CORRESPONDING 
DATA INTO BLOCK", an HQ signal and an SD signal are supplied and a process for extracting the HD 

25 pixels (A to D) and the SO pixels (a to i) In the arrangement shown in Rg. 4 is performed. At step 53 "DATA 
END 7". If data process for all data (for example, one frame) has been completed, the flow advances to 
step 56 "DETERMINE PREDICTED COEFFICIENTS". Otherwise, the flow advances to step 54 "DETER- 
MINE CLASS". 

At step 54 "DETERMINE CLASS", a class is determined corresponding to the pattern of the level 

30 distribution of the SO pixels (a to I) in the vicinity of the HO pixels (A to 0 of Rg. 4) to be predicted. In this 
process, as described atxtve. so as to reduce the number of bits, the SD pixels are compressed by for 
example the ADRC technique. At step 55 "GENERATE NORMAL EQUATIONS", formulas (12). (13). and 
(14). which will t>e descried later, are generated. 

At step 53 "DATA END ?". it is determined whether or not all data has been completely processed. 

35 When all data has been compl'3tely processed, the ftow advances to step 56. At step 56 "DETERMINE 
PREDICTED COEFFICIENTS", formula (14). which will be described later. Is solved by matrix solution so as 
to determine predicted coefficients. At step 57 "STORE PREDICTED COEFFICIENTS", the predicted 
coefficients are stored in the memory. At step 58 "END", the sequence of the steps of the process of the 
learning portion 44 is completed. 

40 Like the atx>ve-described signal converting apparatus, with the pixel arrangement shown in Rg. 4. the 
SD pixels a to i are compressed and encoded. The encoded values are categorized as a class. Predicted 
values of the HD pixels A to 0 are generated by linear combination of the values of the SO pixels a to I and 
the predicted coefficients wi to W„ with a combination shown In Rg. 9. 

Next, the process for extracting predicted coefficients for HO pixels from the values of SD pixels will be 

45 descrit)ed In detail. Assume that the values of SD pixels are xi to Xn and the true value of the considered 
HD pixel is y. For each class, a linear combination of n taps (coefficients wi to Wn) is given as follows. 

y' = wixi + W2X2 + .... + w„x„ (7) 

50 In formula (7), before the learning process Is performed, wj Is an unknown coefficient. 

As described above, the learning process is performed for a plurality of HO data and a plurality of SD 
data for each class. When the number of data is m. formula (7) is modified to formula (8). 

yr = WiXj, + W2Xp + + w,pc,„ (8) 

55 

where j = 1, 2 m. 

When m > n. since wi to Wn are not unique values, elements of an error vector are defined ds follows. 
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In the conventional signal converting apparatus, to use moving information, the motion of an image is 
detected. Moving pixels are processed in a field, whereas still pixels are processed among fields. The 
output signal of each process is mixed with a coefficient corresponding to the amount of motion. Thus, in 
the conventional apparatus, a motion detecting circuit is additionally required. In addition, unless the motion 

5 is accurately detected, image deterioration tends to occur. The ciass categorizing process and the predicted 
value generating process using SD pixels that are three-dimensionally arranged can solve such problems. 

As described atx>ve. the class categorizing process and the predicted value generating process for the 
signal converting apparatus, which converts an SD signal into an HD signal, are the same as those 
processes for the learning portion. Next, the class categorizing process and the predicted value generating 

10 process for SD pixels that are three-dimensionally arranged will be described. First, the relation of positions 
of lines of an HD signal and an SD signal will be explained. 

As with the learning process, when an HD signal is converted into an SD signal, pixels on three 
successive lines of the HD signal are multiplied by non-symmetrical coefficients such as 1/8. 4/8. and 3/8 as 
shown in Fig. 13. one line of the SD signal is formed. These non-symmetrical coefficients are used in the 

/5 reverse order for each field so as to maintain the interlace structure of the SD signal. In other words, if two 
lines of the HD signal are combined and one line of an SD image is formed, since the intervals of lines in 
the n-th field do not accord with the intervals of lines in the (n + 1)-th field, the interiace structure will be 
lost. 

For the class categorizing process and the predicted value generating process, for example 12^SD 
20 pixels that are three-dimensionally distributed are used. There are four patterns of 12 SD pixels con'espond- 
ing to the position of one HD pixel to be predicted. These four patterns are refenred to as mode 1, mode 2. 
mode 3. and mode 4. Figs. 14, 15, 16, and 17 show arrangements of pixels corresponding to mode 1, mode 
2. mode 3. and mode 4, respectively. In Rgs. 14 to Rg. 17, circles represent HD pixels to be predicted and 
squares represent SD pixels to be used for class categorizing process and predicted value generating 
25 process. 

For the simplicity of hardware, in Fig. 15 and 17. SD pixels in the (n - 1)-th field and the (n + 1)-th field 
are partially averaged so as to form SD pixels in the n-th field. The SD pixels in the n field are denoted by 
squares with dotted lines. Figs. 14, 15, 16. and 17 each show a plurality of SD pixels and one HD pixel 
where the SD pixels are used for the class categorizing process and the predicted value generating process 
30 and the HD pixel is to be predicted. 

In mode 1 (arrangement shown in Fig. 14). six SD pixels in the n-th field, three SD pixels in the (n - 1)- 
th field, and three SD pixels in the (n + 1)-th field (a total of 12 SD pixels) are used to perform the class 
categorizing process and the predicted value generating process for an HD pixel. In mode 1 . an HD pixel on 
the tine where a coefficient of 1/8 is multiplied is predicted (see Fig. 13). 
35 In mode 2 (arrangement shown in Fig. 15). SD pixels on the same line in the (n - 1)-th field and tha (n 
+ 1)-th field are averaged so as to interpolate three SD pixels that are required in the n-th field. The, six SD 
pixels including the interpolated SD pixels in the n-th field, three SD pixels other than SD pixels used for 
averaging in the (n - 1)-th field, and three SD pixels other than SD pixels used for averaging in the (n + 1)- 
th field are used. Thus, a total of 12 SD pixels are used. In mode 2. an HD pixel on the line v^e a 
40 coefficient of 4/8 Is multiplied is predicted (see Fig. 13). 

In mode 3 (arrangement of Rg. 16). six SD pixels in the n-th field, four SD pixels in the (n - 1)-th field, 
and two SO pixels in the (n + 1)-th field (a total of 12 SD pixels) are used. In mode 3. an HD pixel on the 
line where a coefficient of 1/8 is multiplied is predicted (see Fig. 13). 

In mode 4 (arangement of Rg. 17). SD pixels in the (n - l)-th field and (n + 1)-th field are averaged so 
45 as to interpolate four SD pixels (two pixels each on two lines) that are required in the n-th field. The eight 
SD pixels including the interpolated SD pixels in the n-th field, two SO pixels other than the SO pixels used 
for averaging in the (n - 1)-th field, and two SD pixels other than the SD pixels used for averaging in the (n 
l)-th field are used. Thus, a total of 12 SD pixels are used. In mode 4, an HD pixel on the line where a 
coefficient of 4/8 is multiplied is predicted (see Fig. 13). 
50 As described above, in modes 3 and mode 4, an HD pixel is generated at the 1/2 position of the 
sampling intervals of SD pixels. In other words, in mode 1 to mode 4, an HD pixel with horizontal and 
vertical resolutions that are twice those of an SD pixel is generated. 

In the learning process, predicted coefficients of each of mode 1 to mode 4 are determined by the 
method of least squares for each class and stored in the memory. Rg. 18 shows a construction of a signal 
55 converting apparatus that generates an output HD image corresponding to an input SO image. 

A signal dO that is an SD image in the (n + 1)-th field, is supplied to an input terminal 61. The dO signal 
is sent to a field memory 62 and up-converting circuits 64a to 64d. The field memory 62 'outputs an SD 
image signal in n-th field. This signal is denoted by signal d1. The signal d1 is sent to the up-converting 
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The formula (16) represents the process for successively storing the values of center of gravity 
obtained in the memory. On the other hand, the formula (17) represents the process for storing the 
cumulated value of the former values of center of gravity in the memory and for dividing the total of the 
cumulated value and the present value of center of gravity by the numt>er of times of cumulation. As 

5 described above, formulas (16) and (17) represent the process for cumulating values normalized with the 
base value base and the dynamic range DR rather than the true value hd of the HO pixel. When the 
cumulated value for each class is stored in the memory and the cumulated value is divided by the number 
of times of cumulation, representative values may be obtained. However, since the cumulated value 
becomes large, the number of bits of the memory may increase. 

10 The value of center of gravity that has been determined and learnt is used for the signal converting 
process. In other words, as with the first embodiment, a class code representing a class is sent to the 
memory. The class code is stored in the memory as an address. The value of center of gravity g(c) of the 
class is read from the memory. With the value of center of gravity g(c), predicted value hd* is generated 
corresponding to formula (18). 

15 

hd' = ORxg(c) + base (18) 

In the first and second embodiments of the present invention, when blocks with dynamic range OR that 
is less than a predetermined value are excluded from data to k>e learnt, the learning process can be 
20 prevented from being affected by noise. 

In addition, according to these embodiments, predicted coefficients arid representative values are pre- 
teamt However, in a high speed system, the predicted coefficients and representative values can t>e 
obtained on real time t>asis. The predicted coefficients and representative values that have been learnt can 
be updated with those obtained on the real time t>asis. ^ 
25 Either the method for predicting the values of all pixels of the output image signal with high resolution 
or the me)hod for predicating the values of pixels that are not present may be selected. 

Moreover, the present invention may t>e applied to image signals other than interlace scanning TV 
signals, sequence scanning TV sigrtals. 

30 Claims 

1. A signal converting apparatus for converting a first digital image signal into a second digital image 
signal having higher resolution than the first digital image signal, comprising: 

class categorizing means for categorizing a considered pixel of the second digital image signal as 
35 a class corresponding to a pattern of level distribution of a plurality of reference pixels of the first digital 
image signal, the reference pixels of the first digital image signal being disposed spatially and/or 
chronologically adjacent to the considered pixel; 

memory means for storing predicted coefficients for each class and for outputting the predicted 
coefficients corresponding to class information received from said class categoriztr^ means; and 
40 predicted value generating means for calculating linear comtunation of the values of the pixels of 

the first digital image signal spatially and/or chronologically adjacent to the considered pixel so as to 
generate a predicted value of the considered pixel. 

2. The signal converting apparatus as set forth in claim 1 , further comprising a learning device for pre- 
45 determining the predicted coefficients. 

wherein said learning device comprises: 

means for forming the first digital image signal with tiie second digital image signal; 
class categorizing means for categorizing a considered pixel of tiie second digital image signal as 
a class corresponding to a pattem of level distribution of a plurality of reference pixels of the first digital 
50 image signal, the reference pixels of the first digital image signal t}eing disposed spatially and/or 
chronologically adjacent to the considered pixel: and 

means for determining predicted coefficients for each class so that the sum of square of an error 
tsetween the true value of the considered pixel and the predicted value becomes minimum when the 
predicted value of the considered pixel is generated by linear combination of a plurality of pixels of the 
55 first digital image signal and the predicted coefficients. 

3. A signal converting apparatus for converting a first digital image signal into a second digital image 
signal having higher resolution than the first digital image signal, comprising: 
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Fig, 10 
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Fig. 13 
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Fig. 16 
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1 2 : BLOCK SEGMENTING CIRCUIT 
1 3 : DATA COMPRESSING CIRCUIT 
15: MEMORY 

1 6 : PREDICTED VALUE GENERATING CIRCUIT 

2 2 : MAXIMUM VALUE/MINIMUM VALUE DETECTING CIRCUIT 
4 4 : LEARNING PORTION 



